Picture conversion apparatus picture conversion method learning apparatus and learning method

ABSTRACT

An apparatus for converting a first picture comprising pixels into a second picture comprising pixels is provided. The second picture is converted by executing on the first picture an adaptive process that determines prediction values of the second picture by using a number of pixel values of the first picture as prediction taps and a number of prediction coefficients that are adapted to the first picture. The apparatus comprises a prediction taps forming circuit for forming a number of prediction taps from the first picture and a picture obtained by the adaptive process and an executing circuit for executing the adaptive process by using the formed number of prediction taps and a number of prediction coefficients that are adapted to the prediction taps. The apparatus further comprises a class taps forming circuit for forming a number of class taps from the first picture and the picture obtained by the adaptive process. The apparatus further comprises a classifying circuit for classifying the number of class taps to determine a class. The executing circuit executes the adaptive process by using the formed number of prediction taps and the number of prediction coefficients corresponding to the class.

BACKGROUND OF THE INVENTION

The present invention relates to a picture conversion apparatus and apicture conversion method. In particular, the invention relates to apicture conversion apparatus and a picture conversion method which makesit possible to obtain a picture having better picture quality.

In converting a standard-resolution or low-resolution picture(hereinafter referred to as an SD (standard definition) picture whereappropriate) into a high-resolution picture (hereinafter-referred to asan HD (high definition) picture where appropriate), or in enlarging apicture, pixel values of absent pixels are interpolated (compensatedfor) by using what is called an interpolation filter or the like.

However, even if pixels are interpolated by using an interpolationfilter, it is difficult to obtain a high-resolution picture because HDpicture components (high-frequency components) that are not included inan SD picture cannot be restored.

In view of the above, the present applicant previously proposed apicture conversion apparatus which converts an SD picture into an HDpicture including high-frequency components that are not included in theSD picture.

In this picture conversion apparatus, high-frequency components that arenot included in an SD picture are restored by executing an adaptiveprocess for determining a prediction value of a pixel of an HD pictureby a linear combination of the SD picture and predetermined predictioncoefficients.

Specifically, for instance, consider the case of determining aprediction value E[y] of a pixel value y of a pixel (hereinafterreferred to as an HD pixel where appropriate) constituting an HD pictureby using a linear first-order combination model that is prescribed bylinear combinations of pixel values (hereinafter referred to as learningdata where appropriate) x₁, x₂, . . . of a certain number of SD pixels(pixels constituting an SD picture) and predetermined predictioncoefficients w₁, w₂, . . . . In this case, a prediction value E[y] canbe expressed by the following formula.

E[y]=w ₁ x ₁ +w ₂ x ₂+. . .  (1)

For generalization, a matrix W that is a set of prediction coefficientsw, a matrix X that is a set of learning data, and a matrix Y′ that is aset of prediction values E[y] are defined as follows: $\begin{matrix}{{x = \begin{pmatrix}x_{11} & x_{12} & \cdots & x_{1n} \\x_{21} & x_{11} & \cdots & x_{2n} \\\cdots & \cdots & \cdots & \cdots \\x_{m1} & x_{m2} & \cdots & x_{mn}\end{pmatrix}}{{W = \begin{pmatrix}W_{1} \\W_{2} \\\cdots \\W_{N}\end{pmatrix}},\quad {Y^{\prime} = \begin{pmatrix}{E\left\lbrack y_{1} \right\rbrack} \\{E\left\lbrack y_{2} \right\rbrack} \\\cdots \\{E\left\lbrack y_{m} \right\rbrack}\end{pmatrix}}}} & \text{(2)}\end{matrix}$

The following observation equation holds:

XW=Y′  (3)

Consider the case of determining prediction values E[y] that are closeto pixel values y of HD pixels by applying a least squared method tothis observation equation. In this case, a matrix Y that is a set oftrue pixel values y of HD pixels as teacher data and a matrix E that isa set of residuals e of prediction values E[y] with respect to the pixelvalues y of the HD pixels are defined as follows: $\begin{matrix}{{E = \begin{pmatrix}e_{1} \\e_{2} \\\cdots \\e_{m}\end{pmatrix}},\quad {Y^{\prime} = \begin{pmatrix}y_{1} \\y_{2} \\\cdots \\y_{m}\end{pmatrix}}} & \text{(4)}\end{matrix}$

From Formula (3), the following residual equation holds:

XW=Y+E  (5)

In this case, prediction coefficients w_(i) for determining predictionvalues E[y] that are close to the pixel values y of the HD pixels aredetermined by minimizing the following squared error: $\begin{matrix}{\sum\limits_{i = 1}^{m}e_{i}^{2}} & \text{(6)}\end{matrix}$

Therefore, prediction coefficients w_(i) that satisfy the followingequations (derivatives of the above squared error with respect to theprediction coefficients w_(i) are 0) are optimum values for determiningprediction values E[y] close to the pixel values y of the HD pixels.$\begin{matrix}{{{e_{1}\frac{\partial e_{1}}{\partial w_{i}}} + {e_{2}\frac{\partial e_{2}}{\partial w_{i}}} + \quad \ldots \quad + {e_{m}\frac{\partial e_{m}}{\partial w_{i}}}} = {0\left( {{i = 1},2,\ldots \quad,n} \right)}} & \text{(7)}\end{matrix}$

In view of the above, first, the following equations are obtained bydifferentiating Formula (5) with respect to prediction coefficientsw_(i). $\begin{matrix}{{\frac{\partial e_{i}}{\partial w_{1}} = x_{i1}},{\frac{\partial e_{i}}{\partial w_{2}} = x_{i2}},\ldots \quad,\quad {\frac{\partial e_{i}}{\partial w_{n}} = x_{m}},\left( {{i = 1},2,\ldots \quad,m} \right)} & \text{(8)}\end{matrix}$

Formula (9) is obtained from Formula (7) and (8). $\begin{matrix}{{{\sum\limits_{i = 1}^{m}{e_{i}x_{i1}}} = 0},{{\sum\limits_{i = 1}^{m}{e_{i}x_{i2}}} = 0},{{\ldots \quad {\sum\limits_{i = 1}^{m}{e_{i}x_{in}}}} = 0}} & \text{(9)}\end{matrix}$

By considering the relationship between the learning data x, theprediction coefficients w, the teacher data y, and the residuals e inthe residual equation of Formula (5), the following normal equations canbe obtained from Formula (9): $\begin{matrix}\begin{Bmatrix}\left. {{{\left( {\sum\limits_{i = 1}^{m}{x_{i1}x_{i1}}} \right)w_{1}} + {\left( {\sum\limits_{i = 1}^{m}{x_{i1}x_{i2}}} \right)w_{2}} + \ldots \quad + {\left( {\sum\limits_{i = 1}^{m}{x_{i1}x_{in}}} \right)w_{n}}} = {\sum\limits_{i = 1}^{m}{x_{i1}y_{i}}}} \right) \\\left. {{{\left( {\sum\limits_{i = 1}^{m}{x_{i2}x_{i1}}} \right)w_{1}} + {\left( {\sum\limits_{i = 1}^{m}{x_{i2}x_{i2}}} \right)w_{2}} + \ldots \quad + {\left( {\sum\limits_{i = 1}^{m}{x_{i2}x_{in}}} \right)w_{n}}} = {\sum\limits_{i = 1}^{m}{x_{i2}y_{i}}}} \right) \\\left. {{{\left( {\sum\limits_{i = 1}^{m}{x_{in}x_{i1}}} \right)w_{1}} + {\left( {\sum\limits_{i = 1}^{m}{xnx}_{i2}} \right)w_{2}} + \ldots \quad + {\left( {\sum\limits_{i = 1}^{m}{x_{in}x_{in}}} \right)w_{n}}} = {\sum\limits_{i = 1}^{m}{x_{in}y_{i}}}} \right)\end{Bmatrix} & \text{(10)}\end{matrix}$

The normal equations of Formula (10) can be obtained in the same numberas the number of prediction coefficients w to be determined. Therefore,optimum prediction coefficients w can be determined by solving Formula(10) (for Formula (10) to be soluble, the matrix of the coefficients ofthe prediction coefficients w need to be regular). To solve Formula(10), it is possible to use a sweep-out method (Gauss-Jordan eliminationmethod) or the like.

The adaptive process is a process for determining optimum predictioncoefficients w in the above manner and then determining predictionvalues E[y] that are close to the component signals y according toFormula (1) by using the optimum prediction values w (the adaptiveprocess includes a case of determining prediction coefficients w inadvance and determining prediction values by using the predictioncoefficients w).

The adaptive process is different from the interpolation process in thatcomponents not included in an SD picture but included in an HD pictureare reproduced. That is, the adaptive process appears the same as theinterpolation process using an interpolation filter, for instance, aslong as only Formula (1) is concerned. However, the adaptive process canreproduce components in an HD picture because prediction coefficients wcorresponding to tap coefficients of the interpolation filter aredetermined by what is called learning by using teacher-data y. That is,a high-resolution picture can be obtained easily. From this fact, it canbe said that the adaptive process is a process having a function ofcreating resolution of a picture.

FIG. 9 is an example of a configuration of a picture conversionapparatus which converts an SD picture as a digital signal into an HDpicture.

An SD picture is supplied to a delay line 107, and blocking circuits 1and 2. The SD picture is delayed by, for instance, one frame by thedelay line 107 and then supplied to the blocking circuits 1 and 2.Therefore, the blocking circuits 1 and 2 are supplied with an SD pictureof a current frame (hereinafter referred to as a subject frame whereappropriate) as a subject of conversion into an HD picture and an SDpicture of a 1-frame preceding frame (hereinafter referred to as apreceding frame where appropriate).

In the blocking circuit 1 or 2, HD pixels which constitute an HD pictureof the subject frame are sequentially employed as the subject pixel andprediction taps or class taps for the subject pixel are formed from theSD pictures of the subject frame and the preceding frame.

It is assumed here that, for example, HD pixels and SD pixels have arelationship as shown in FIG. 10. That is, in this case, one SD pixel(indicated by mark “□”, in the figure) corresponds to four HD pixels(indicated by mark “◯” in the figure) located at top-left, top-right,bottom-left, and bottom-right positions of the SD pixel and adjacentthereto. Therefore, the SD pixels are pixels obtained by decimating theHD pixels at a rate of one for two in both horizontal and verticaldirections.

In the blocking circuit 1 or 2, for example, when some HD pixel isemployed as the subject pixel, a block (processing block) of 3×3 pixels(horizontal/vertical) having, as the center, the SD pixel in the subjectframe corresponding to the subject pixel (HD pixel) is formed as shownin FIG. 10 and a block of 3×3 pixels having, as the center, the SD pixelin the preceding frame corresponding to the subject pixel is formed asshown in FIG. 11. The 18 pixels (SD pixels) in total are employed asprediction taps or class taps. FIG. 10 shows SD pixels and HD pixels inthe subject frame by marks “□” and “◯” respectively, and FIG. 11 showsSD pixels and HD pixels in the preceding frame by marks “▪” and “”respectively.

The prediction taps obtained by the blocking circuit 1 are supplied to aprediction operation circuit 6, and the class taps obtained by theblocking circuit 2 are supplied to a class code generation circuit 4 viaan ADRC circuit 3.

In the above case, prediction taps and class taps are formed by 3×3 SDpixels having, as the center, the SD pixel in the subject framecorresponding to the subject pixel and 3×3 pixels having, as the center,the SD pixel in the preceding frame corresponding to the subject pixel.Therefore, the same prediction taps and class taps are formed when anyof HD pixels a, b, c, and d shown in FIG. 10 is employed as the subjectpixel.

The class taps that have been supplied to the class code generationcircuit 4 via the ADRC circuit 3 are classified there. That is, theclass code generation circuit 4 outputs, as a class of the class taps(or subject pixel), a value corresponding to a pattern of pixel valuesof the SD pixels (as described above, 18 SD pixels) that constitute theclass taps.

Where a large number of bits, for instance, 8 bits, are allocated torepresent the pixel value of each SD pixel, the number of patterns ofpixel values of 18 SD pixels is enormous, that is, (2⁸)¹⁸, making itdifficult to increase the speed of the following process.

In view of the above, for example, an ADRC (adaptive dynamic rangecoding) process that is a process for decreasing the number of bits ofSD pixels constituting class taps is executed on the class taps in theADRC circuit 3 as a pre-process for the classification.

Specifically, in the ADRC circuit 3, first a pixel having the maximumpixel value (hereinafter referred to as a maximum pixel whereappropriate) and a pixel having the minimum pixel value (hereinafterreferred to as a minimum pixel where appropriate) among the 18 SD pixelsconstituting the class taps are detected. The difference DR (=MAX−MIN)between the pixel value MAX of the maximum pixel and the pixel value MINof the minimum pixel is calculated and employed as a local dynamic rangeof the processing block. Based on the dynamic range DR, the respectivepixel values constituting the processing block are re-quantized into Kbits that are smaller than the originally allocated number of bits. Thatis, the respective pixel values constituting the processing block aresubtracted by the pixel value MIN of the minimum pixel and resultingdifferences are divided by DR/2^(K).

As a result, the respective pixel values constituting the processingblock are expressed by K bits. Therefore, where, for example, K=1, thenumber of patterns of pixel values of 18 SD pixels become (2¹)¹⁸, whichis much smaller than in the case where the ADRC process is not executed.The ADRC process for causing pixel values to be expressed by K bits willbe hereinafter referred to as a K-bit ADRC process, where appropriate.

The class code generation circuit 4 executes a classification process onthe class taps that have been subjected to the above ADRC process,whereby a value corresponding to a pattern of the SD pixel valuesconstituting the class taps is supplied to a prediction coefficientsmemory 5 as a class of the class taps (subject pixel corresponding toit).

The prediction coefficients memory 5 stores, for each class, predictioncoefficients that have been determined in advance through learning. Whensupplied with a class from the class code generation circuit 4, theprediction coefficients memory S reads out prediction coefficients thatare stored at an address corresponding to the class and supplies thoseto the prediction operation circuit 6.

In the prediction operation circuit 6, the operation represented byFormula (1), that is, an adaptive process, is performed by usingprediction taps (pixel values of SD pixels constituting the predictiontaps) x₁, x₂, . . . that are supplied from the blocking circuit 2 andprediction coefficients adapted to the prediction taps, that is,prediction coefficients w₁, w₂, . . . corresponding to the class of thesubject pixel that are supplied from the prediction coefficients memory5. A prediction value E[y] of the subject pixel y is thereby determinedand output as a pixel value of the subject pixel (HD pixel).

Thereafter, similar processes are sequentially executed while the otherHD pixels of the subject frame are employed as the subject pixel,whereby the SD picture is converted into an HD picture.

FIG. 12 shows an example of a configuration of a learning apparatuswhich executes a learning process for calculating predictioncoefficients to be stored in the prediction coefficients memory 5 shownin FIG. 9.

An HD picture (HD picture for learning) to serve as teacher data y oflearning is supplied to a decimation circuit 101 and a teacher dataextraction circuit 27. In the decimation circuit 101, the HD pixel isreduced in the number of pixels by decimation and is thereby convertedinto an SD picture (SD picture for learning). Specifically, since one SDpixel corresponds to four HD pixels adjacent thereto as described abovein connection with FIG. 10, for example, in the decimation circuit 101the HD picture is divided into blocks of 2×2 HD pixels and the averagevalue of those pixels is employed as a pixel value of the SD pixellocated at the center of each block of 2×2 HD pixels (i.e., the SD pixelcorresponding to the 2×2 HD pixels).

The SD picture obtained by the decimation circuit 101 is supplied to adelay line 128 and blocking circuits 21 and 22.

The delay line 128, the blocking circuits 21 and 22, an ADRC circuit 23,and a class code generation circuit 24 execute the same processes as theblocking circuits 1 and 2, the ADRC circuit 3, and the class codegeneration circuit 4 shown in FIG. 9, respectively. As a result, theblocking circuit 21 outputs prediction taps that have been formed forthe subject pixel and the class code generation circuit 24 outputs aclass of the subject pixel.

The class that is output from the class code generation circuit 24 issupplied to respective address terminals (AD) of a prediction tapsmemory 25 and a teacher data memory 26. The prediction taps that areoutput from the blocking circuit 21 are supplied to the prediction tapsmemory 25. The prediction taps memory 25 stores, as learning data, theprediction taps that are supplied from the blocking circuit 21 at anaddress corresponding to the class that is supplied from the class codegeneration circuit 24.

On the other hand, a teacher data extraction circuit 27 extracts an HDpixel as the subject pixel from an HD picture supplied thereto, andsupplies it to the teacher data memory 26 as teacher data. The teacherdata memory 26 stores the teacher data that is supplied from the teacherdata extraction circuit 27 at an address corresponding to the class thatis supplied from the class code generation circuit 24.

Thereafter, similar processes are executed until all HD pixelsconstituting the HD picture that is prepared for the learning in advanceare employed as the subject pixel.

As a result, SD pixels and an HD pixel that are in the positionalrelationships described above in connection with FIGS. 10 and 11 arestored, as learning data x and teacher data y, at the same addresses ofthe prediction taps memory 25 and the teacher data memory 26,respectively.

The prediction taps memory 25 and the teacher data memory 26 can storeplural pieces of information at the same address, whereby a plurality oflearning data x and a plurality of teacher data y that are classified asthe same class can be stored at the same addresses.

Then, an operation circuit 29 reads out, from the prediction taps memory25 and the teacher data memory 26, pixel values of SD pixelsconstituting prediction taps as learning data and pixel values of HDpixels as teacher data that are stored at the same addresses,respectively. The operation circuit 29 calculates predictioncoefficients that minimize errors between prediction values and theteacher data by a least squares method by using those pixel values. Thatis, the operation circuit 29 establishes normal equations of Formula(10) for each class and determines prediction coefficients for eachclass by solving the normal equations.

The prediction coefficients for the respective classes that have beendetermined by the operation circuit 29 in the above manner are stored inthe prediction coefficients memory 5 shown in FIG. 9 at an addresscorresponding to the class.

Where prediction coefficients obtained by the learning apparatus of FIG.12 are stored in the prediction coefficients memory 5 of the pictureconversion apparatus of FIG. 9 and a conversion from an SD picture to anHD picture is performed, basically the picture quality of a resulting HDpicture can be improved by increasing the number of SD pixelsconstituting class taps and prediction taps.

However, as the number of SD pixels constituting class taps andprediction taps is increased, the SD pixels come to include ones thatare distant from the subject pixel spatially or temporally. In such acase, SD pixels having no correlation with the subject pixel come to beincluded in class taps and prediction taps. Once this situation occurs,it is difficult to improve the picture quality of an HD picture byfurther adding SD pixels having no correlation to class taps andprediction taps.

OBJECTS OF THE INVENTION

The present invention has been made in view of the above circumstances,and an object of the invention is to make it possible to further improvethe picture quality of an HD picture.

SUMMARY OF THE INVENTION

In order to attain the above objects, according to an aspect of thepresent invention, an apparatus for converting a first picturecomprising pixels into a second picture comprising pixels is provided.The second picture is converted by executing on the first picture anadaptive process that determines prediction values of the second pictureby using a number of pixel values of the first picture as predictiontaps and a number of prediction coefficients that are adapted to thefirst picture. The apparatus comprises a prediction taps forming circuitfor forming a number of prediction taps from the first picture and athird picture obtained by the adaptive process and an executing circuitfor executing the adaptive process by using the formed number ofprediction taps and a number of prediction coefficients that are adaptedto the prediction taps.

The apparatus further comprises a class taps forming circuit for forminga number of class taps from the first picture and the picture obtainedby the adaptive process. The apparatus further comprises a classifyingcircuit for classifying the number of class taps to determine a class.The executing circuit executes the adaptive process by using the formednumber of prediction taps and the number of prediction coefficientscorresponding to the class.

Further, the execution circuit reads out the number of predictioncoefficients from a memory in response to the class and calculating theprediction value of the second picture by using the formed number ofprediction taps and the read number of prediction coefficients.

Further, the memory stores a number of prediction coefficients forrespective classes.

Further, the number of prediction coefficients for respective classesare generated in advance by using a second learning picture having aquality corresponding to a quality of the second picture.

Further, the number of prediction coefficients for respective classesare chosen so as to minimize an error between the second learningpicture and a picture predicted from a first learning picture and thesecond learning picture.

Further, the number of prediction coefficients for respective classesare selected so as to minimize an error between the second learningpicture and a picture predicted from a first learning picture and anadaptive processed second picture, the adaptive processed second picturebeing obtained by executing the adaptive process on the first learningpicture.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the invention, reference is made tothe following description and accompanying drawings, in which:

FIG. 1 is a block diagram showing an example configuration of anembodiment of a picture conversion apparatus to which the presentinvention is applied;

FIG. 2 is a chart showing a frame relationship between an SD picture andan HD picture that are input to blocking circuits 1 and 2 shown in FIG.1;

FIG. 3 is a chart showing a relationship between SD pixels and HDpixels;

FIG. 4 is a chart for description of processes of the blocking circuits1 and 2 shown in FIG. 1;

FIG. 5 is a block diagram showing an example configuration of a firstembodiment of a learning apparatus to which the present invention isapplied;

FIG. 6 is a block diagram showing an example configuration of a learningsection 13 shown in FIG. 5;

FIG. 7 is a block diagram showing an example configuration of a secondembodiment of a learning apparatus to which the present invention isapplied;

FIG. 8 is a block diagram showing an example configuration of a learningsection 36 shown in FIG. 7;

FIG. 9 is a block diagram showing an example configuration of a pictureconversion apparatus that was previously proposed by the presentapplicant;

FIG. 10 is a chart for description of processes of blocking circuits 1and 2 shown in FIG. 9;

FIG. 11 is a chart for description of the processes of blocking circuits1 and 2 shown in FIG. 9; and

FIG. 12 is a block diagram showing an example configuration of alearning apparatus which executes a learning process for determiningprediction coefficients to be stored in a prediction coefficients memory5 shown in FIG. 9.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The embodiments of the present invention will be described below. FIG. 1shows an example configuration of an embodiment of a picture conversionapparatus to which the present invention is applied. In FIG. 1,components having the corresponding components in FIG. 9 are given thesame reference numerals as the latter. That is, this picture conversionapparatus is configured basically in the same manner as the pictureconversion apparatus of FIG. 9 except that instead of the delay line 107a delay line 7 is provided which delays an output picture of theprediction operation circuit 6 and supplies a delayed picture to theblocking circuits 1 and 2.

The delay line 7 delays an HD picture that is output from the predictionoperation circuit 6 by, for instance, one frame and supplies a delayedHD picture to the blocking circuits 1 and 2. Therefore, in theembodiment of FIG. 1, as shown in FIG. 2, at the same time as an SDpicture of an Nth frame is supplied to the blocking circuits 1 and 2, anHD picture of an (N−1)th frame (i.e., a frame that precedes the Nthframe by one frame) that has been obtained by an adaptive process issupplied thereto.

In the above-configured picture conversion apparatus, as describedabove, an SD picture of the subject frame and an HD picture of thepreceding frame are supplied to the blocking circuits 1 and 2. That is,in contrast to the configuration of FIG. 9 in which SD picture of thesubject frame and the preceding frame are supplied, in this embodimentan HD picture that has already been obtained by the prediction operationcircuit 6 through an adaptive process is supplied together with an SDpicture of the subject frame.

In the blocking circuit 1 or 2, while HD pixels constituting an HDpicture of the subject frame are sequentially employed as the subjectpixel, prediction taps or class taps for the subject pixel are formedfrom an SD picture of the subject frame and an HD picture of thepreceding frame.

Also in this case, it is assumed that, for example, HD pixels and SDpixels have a relationship as shown in FIG. 3 that is the same as theabove-described relationship shown in FIG. 10. That is, in this case,one SD pixel corresponds to four HD pixels located at top-left,top-right, bottom-left, and bottom-right positions of the SD pixel andadjacent thereto. Therefore, the SD pixels are pixels obtained bydecimating the HD pixels at a rate of one for two in both the horizontaland vertical directions.

In the blocking circuit 1 or 2, for example, when some HD pixel isemployed as the subject pixel, a block (processing block) of 3×3 SDpixels (indicated by mark “□” in FIG. 3) having, as the center, the SDpixel in the subject frame corresponding to the subject pixel (HD pixel)is formed as enclosed by a broken line in FIG. 3, and a block of HDpixels corresponding to 3×3 (=9) respective SD pixels having, as thecenter, the SD pixel in the preceding frame corresponding to the subjectpixel, that is, 6×6 (=36) HD pixels encleosed by a broken line in FIG. 4(since four HD pixels correspond to one SD pixel, the number of HDpixels corresponding to the 9 respective SD pixels is 4×=36; indicatedby mark “” in FIG. 4) is formed. The 45 pixels (9 SD pixels and 36 HDpixels) in total are employed as prediction taps or class taps.

Like FIG. 10, FIG. 3 shows SD pixels and HD pixels in the subject frameby marks “□” and “◯” respectively. FIG. 4 shows SD pixels in the subjectframe and HD pixels in the preceding frame by marks “▪” and “”respectively.

The prediction taps obtained by the blocking circuit 1 are supplied tothe prediction operation circuit 6, and the class taps obtained by theblocking circuit 2 are supplied to the ADRC circuit 3.

In the above case, to simplify the description, prediction taps andclass taps are formed by the same SD pixels and HD pixels for a certainsubject pixel. However, it is not always necessary to form those kindsof taps by the same SD pixels and HD pixels. That is, the number ofprediction taps can be changed to the number of class taps.

Although in the above case prediction taps and class taps are formed by3×3 SD pixels in a square region in the subject frame and 6×6 HD pixelsin a square region in the preceding frame, they can be formed by SDpixels and HD pixels in a region having some other shape such as arectangle, a cruciform or a rhombus. It is even possible to use pixelsconstituting an SD picture and an HD picture of a frame that precedesthe preceding frame by one frame or a frame that succeeds the subjectframe by one frame.

It is preferable to form prediction taps and class taps by SD pixels andHD pixels having strong correlations with the subject pixel. Therefore,basically it is preferable to use SD pixels and HD pixels that are closeto the subject pixel spatially and/or temporally.

In the above case, prediction taps and class taps are formed by 3×3 SDpixels having, as the center, the SD pixel in the subject framecorresponding to the subject pixel and a total of 36 HD pixelscorresponding to 3×3 respective SD pixels having, as the center, the SDpixel in the preceding frame corresponding to the subject pixel.Therefore, as described above in connection with FIG. 10, the sameprediction taps and class taps are formed when any of four HD pixelscorresponding to some SD pixel is employed as the subject pixel (it ispossible to form different prediction taps and class taps).

The ADRC circuit 3 executes, for instance, a 1-bit ADRC process on classtaps that are supplied from the blocking circuit 2, and resulting classtaps are supplied to the class code generation circuit 4. The class codegeneration circuit 4 outputs, as a class of those class taps (or thesubject pixel), a value corresponding to a pattern of pixel values of SDpixels and HD pixels constituting the class taps.

The class that is output from the class code generation circuit 4 issupplied to the prediction coefficients memory 5 as an address.

The prediction coefficients memory 5 stores, for each class, predictioncoefficients that have been determined through learning (describedlater). When supplied with a class from the class code generationcircuit 4, the prediction coefficients memory 5 reads out predictioncoefficients that are stored at an address corresponding to the classand supplies those to the prediction operation circuit 6. Specifically,where prediction taps are constituted of m SD pixels x₁, x₂, . . . ,x_(m) and n HD pixels y₁, Y₂, y_(n) (in this embodiment, m=9 and n=36),the prediction coefficients memory 5 stores prediction coefficientsw_(x1), w_(x2), . . . , w_(xm) to be multiplied by the respective SDpixels x₁, x₂, . . . , x_(m) and prediction coefficients w_(y1), w_(y2),. . . , w_(yn) to be multiplied by the respective HD pixels y₁, y₂, . .. , y_(n). The prediction coefficients memory 5 supplies the predictionoperation circuit 6 with prediction coefficients w_(x1), w_(x2), . . . ,w_(xm) and w_(y1), w_(y2), . . . , w_(yn) that are stored at an addresscorresponding to the class that is supplied from the class codegeneration circuit 4.

The prediction operation circuit 6 performs an operation of Formula (11)that corresponds to Formula (1), that is, an adaptive process by usingprediction taps (i.e., pixel values of SD pixels and HD pixelsconstituting prediction taps) x₁, x₂, . . . , x_(m) and y₁, y₂, . . . ,y_(n) and prediction coefficients adapted to the prediction taps, thatis, prediction coefficients w_(x1), W_(x2), . . . , w_(xm) and w_(y1),w_(y2), . . . , w_(yn) that are supplied from the predictioncoefficients memory 5 and correspond to the class of the subject pixel.As a result, a prediction value E[z] is determined and output as a pixelvalue of the subject pixel (HD pixel). $\begin{matrix}\begin{matrix}{{E\lbrack z\rbrack} = \quad {{w_{x1}x_{1}} + {w_{x2}x_{2}} + \cdots \quad + {w_{xm}x_{m}} +}} \\{\quad {{w_{y1}y_{1}} + {w_{y2}y_{2}} + \cdots + {w_{yn}y_{n}}}}\end{matrix} & (11)\end{matrix}$

That is, while in the case of FIG. 9 the adaptive process is executed byusing only SD pixels, in this embodiment the adaptive process isexecuted by using also HD pixels (in this case, HD pixels constitutingan HD picture of the preceding frame that has been obtained by thepreviously executed adaptive process).

Now, the adaptive process that was described above by using Formulae(1)-(7) are extended so as to use not only SD pixels but also HD pixels.

For example, now we consider a case of determining a prediction valueE[z] of a pixel value z of an HD pixel of the subject frame by using alinear first-order combination model that is prescribed by a linearcombination of pixel values x₁, x₂, . . . , x_(m) of several SD pixelsof the subject frame and pixel values y₁, y₂, . . . , y_(n) of severalHD pixels of the preceding frame and prediction coefficients w_(x1),w_(x2), . . . , w_(xm) and w_(y1), w_(y2), . . . , w_(yn). In this case,the prediction value E[z] of the pixel value z is expressed by the aboveFormula (11).

Now, k HD pixels z₁, z₂, . . . , z_(k) of the subject frame are preparedfor learning of prediction coefficients w_(x1), w_(x2), . . . , w_(xm)and w_(y1), w_(y2), . . . , w_(yn). Where each of the k HD pixels areemployed as the subject pixel, k sets of SD pixels x₁, x₂, . . . , x_(m)of the subject frame and HD pixels y₁, y₂, . . . , y_(n) of thepreceding frame that constitute prediction taps are prepared.

When an HD pixel z_(j) of the subject frame is employed as the subjectpixel, a set of SD pixels of the subject frame and HD pixels of thepreceding frame that constitute prediction taps are expressed as x_(j1),x_(j2), . . . , x_(jm) and y_(j1), y_(j2), . . . , y_(jn), respectively.In this case, Formula (12) holds. $\begin{matrix}\begin{matrix}\begin{matrix}{{{E\left\lbrack z_{j} \right\rbrack} = \quad {{w_{x1}x_{j1}} + {w_{x2}x_{j2}} + \cdots \quad + {w_{xm}x_{jm}} +}}\quad} \\{\quad {{w_{y1}y\quad {j1}} + {w_{y2}y_{j2}} + \cdots + {w_{yn}y_{jn}}}}\end{matrix} \\{{\text{where~~~}\text{j}\text{= 1, 2,~~. . .~~,}}\quad {k.}}\end{matrix} & (12)\end{matrix}$

Now, with an assumption that k≧m+n (for normal equations of Formula (18)(described later) to be soluble, the number k of data used in thelearning should be m+n or more), we consider a case of determining aprediction value E[zj] that is close to the true value z_(j) fromFormula (12) (or Formula (11)) by a least squares method.

In this case, an error e_(j) of a prediction value E[zj] with respect tothe true value zj is expressed by the following formula. $\begin{matrix}\begin{matrix}{e_{j} = \quad {z_{j} - {e\left\lbrack z_{j} \right\rbrack}}} \\{= \quad {z_{j} - \left( {{w_{x1}x_{j1}} + {w_{x2}x_{j2}} + \quad \cdots + {w_{xm}x_{jm}} +} \right.}} \\\left. \quad {{w_{y1}y_{j1}} + {w_{y2}y_{j2}} + \quad \cdots + {w_{yn}y_{jn}}} \right)\end{matrix} & (13)\end{matrix}$

Therefore, if prediction coefficients w_(x1), w_(x2), . . . , w_(xm) andw_(y1), w_(y2), . . . , w_(yn) that minimize Formula (14) are obtained,they are a set of prediction coefficients that minimizes a squared errorE² of prediction values E [zj] with respect to the true values z_(j).$\begin{matrix}{E^{2} = {\sum\limits_{j = 1}^{k}\left\{ e_{j} \right\}^{2}}} & (14)\end{matrix}$

That is, prediction coefficients w that zero derivatives of the squarederror E² that is expressed by Formula (14) with respect to theprediction coefficients w are optimum values for determining aprediction value E[z] that is close to the pixel value z of the HDpixel.

In view of the above, partially differentiating Formula (14) withrespect to the prediction coefficients w_(xi) (i=1, 2, . . . , m) andw_(yi) (i=1, 2, . . . , n), we obtain Formulae (15) and (16),respectively. $\begin{matrix}{\frac{\partial E^{2}}{\partial W_{xi}} = {{\sum\limits_{j = 1}^{k}{2\left( \frac{\partial e_{j}}{\partial w_{xi}} \right)e_{j}}} = {\sum\limits_{j = 1}^{k}{2{x_{ji} \cdot e_{j}}}}}} & (15) \\{\frac{\partial E^{2}}{\partial W_{xi}} = {{\sum\limits_{j = 1}^{k}{2\left( \frac{\partial e_{j}}{\partial w_{yi}} \right)e_{j}}} = {\sum\limits_{j = 1}^{k}{2{y_{ji} \cdot e_{j}}}}}} & (16)\end{matrix}$

Since prediction coefficients w_(xi) and w_(yi) that make Formula (15)and (16) zero should be determined, normal equations of Formula (18) areobtained by defining X_(ji), Y_(ji), and Z_(i) as Formula (17).$\begin{matrix}{{X_{ji} = {\sum\limits_{p = 1}^{k}{x_{pi} \cdot x_{pj}}}}{X_{ji} = {\sum\limits_{p = 1}^{k}{x_{pi} \cdot x_{pj}}}}{Z_{i} = {\sum\limits_{p = 1}^{k}{x_{ij} \cdot y_{j}}}}} & (17) \\{{\left\lbrack {\begin{matrix}X_{11} & X_{12} & \cdots & X_{1m} & Y_{11} & Y_{12} & \cdots \\X_{21} & X_{22} & \cdots & X_{2m} & Y_{21} & Y_{22} & \cdots \\\quad & \quad & \quad & \quad & \quad & \quad & \quad \\\cdots & \cdots & \cdots & \cdots & \quad & \quad & \quad \\\quad & \quad & \quad & \quad & \quad & \quad & \quad \\\quad & \quad & \quad & \quad & \quad & \quad & \quad \\X_{k1} & X_{k2} & \cdots & X_{km} & Y_{k1} & Y_{k2} & \cdots\end{matrix}\begin{matrix}Y_{1n} \\Y_{2n} \\\quad \\\quad \\\quad \\\quad \\Y_{kn}\end{matrix}} \right\rbrack \begin{bmatrix}W_{x1} \\W_{x2} \\\vdots \\W_{xn} \\W_{y1} \\\vdots \\W_{yn}\end{bmatrix}} = \begin{bmatrix}Z_{1} \\Z_{2} \\\quad \\\vdots \\\quad \\\quad \\Z_{k}\end{bmatrix}} & (18)\end{matrix}$

The normal equations of Formula (18) can be solved by using, forinstance, a sweep-out method as in the case of solving theabove-described Formula (10).

Prediction coefficients w_(x1), w_(x2), . . . , w_(xm) and w_(y1),w_(y2), . . . , w_(yn) that have been obtained by establishing thenormal equations of Formula (18) for each class are stored in theprediction coefficients memory 5. The prediction operation circuit 6determined a prediction value E[z] close to the true value z byperforming the operation of Formula (11) using prediction coefficientsw_(x1), w_(x2), . . . , w_(xm) and w_(y1), w_(y2), . . . , w_(yn)corresponding to the class of the subject pixel and SD pixels x₁, x₂, .. . , x_(m) and HD pixels y₁, y₂, . . . , y_(n) constituting predictiontaps that have been formed for the subject pixel.

Similar processes are sequentially executed while the other HD pixels ofthe subject frame are employed as the subject pixel, whereby the SDpixels are converted into HD pixels.

Since as described above class taps and prediction taps are formed byusing not only SD pixels but also HD pixels, the class taps and theprediction taps can include many pixels that are close to the subjectpixel spatially and/or temporally. Further, since HD pixels in whichhigh-frequency components have been restored through the adaptiveprocess are used in forming class taps and prediction taps, the qualityof a resulting HD picture can be improved as compared to the case offorming class taps and prediction taps using only SD pixels.

Formulas (15) and (16) correspond to the above-described Formula (8) andFormula (18) corresponds to the above-described Formula (10). Formula(15) and (16) are formula extended from Formula (8) in that Formula (16)is newly introduced. Formula (18) is a formula extended from Formula(10) in that the portions enclosed by broken lines in Formula (15) areadded.

In the embodiment of FIG. 1, no HD picture has been output from theprediction operation circuit 6 at a time point when an SD picture of thefirst frame is input. At this time point, the blocking circuits 1 and 2executes processes using, for instance, predetermined initial values orindefinite values instead of an HD picture from the prediction operationcircuit 6.

FIG. 5 shows an example configuration of a first embodiment of alearning apparatus which executes a learning process for calculatingprediction coefficients for each class to be stored in the predictioncoefficients memory 5 shown in FIG. 1.

A HD picture of a digital signal (HD signal for learning) to becometeacher data z in learning is supplied to vertical decimation filter 11and a learning section 13. In the vertical decimation circuit 11, thenumber of pixels of the HD pixels for learning is, for instance, halvedin the vertical direction and a resulting picture is supplied to ahorizontal decimation filter 12. In the horizontal decimation filter 12,the number of pixels of the output of the vertical decimation filter 12is, for instance, halved in the horizontal direction. As a result, an SDpicture for learning that is constituted of SD pixels indicated by mark“□” in FIG. 3 is formed from the HD picture for learning that isconstituted of HD pixels indicated by mark “◯” in FIG. 3. Specifically,for example, the vertical decimation filter 11 and the horizontaldecimation filter 12 equivalently execute a process that average valuesare calculated for the HD picture for learning in units of 2×2 HD pixelsand employed as a pixel value of an SD pixel located at the center ofeach set of 2×2 HD pixels.

The SD picture that is output from the horizontal decimation filter 12is supplied to the learning section 13.

The learning section 13 determines, for each class, predictioncoefficients that decrease errors of a picture obtained by a linearcombination of the prediction coefficients and the SD picture and the HDpicture for learning that are input to the learning section 13 withrespect to the HD picture for learning by establishing and solvingnormal equations of the above Formula (18). The prediction coefficientsfor the respective classes are supplied to a prediction coefficientsmemory 14 together with the classes. The prediction coefficients memory14 stores the prediction coefficients supplied from the learning section13 at respective addresses corresponding to the classes also suppliedfrom the learning section 13. As a result, the prediction coefficientsfor the respective classes are stored in the prediction coefficientsmemory 14.

FIG. 6 shows an example configuration of the learning section 13 shownin FIG. 5. In FIG. 6, components having the corresponding components inthe learning apparatus of FIG. 12 are given the same reference numeralsas the latter. That is, the learning section 13 is configured basicallyin the same manner as the learning apparatus of FIG. 12 except that thedecimation circuit 101 is not provided and a delay line 28 is providedinstead of the delay line 128.

An SD picture for learning is supplied to the blocking circuits 21 and22 and an HD picture for learning is supplied to the teacher dataextraction circuit 27 and the delay line 28. The delay line 28 delaysthe received HD picture by, for instance, a time corresponding to oneframe and supplies a delayed HD picture to the blocking circuits 21 and22. Therefore, the blocking circuits 21 and 22 are supplied with both ofan SD picture of the subject frame and an HD picture of the precedingframe.

The blocking circuits 21 and 22, the ADRC circuit 23, and the class codegeneration circuit 24 execute the same processes as the blockingcircuits 1 and 2, the ADRC circuit 3, and the class code generationcircuit 4 shown in FIG. 9, respectively. As a result, the blockingcircuit 21 outputs prediction taps that have been formed for the subjectpixel and the class code generation circuit 24 outputs a class of thesubject pixel.

The class that is output from the class code generation circuit 24 issupplied to respective address terminals (AD) of the prediction tapsmemory 25 and the teacher data memory 26. The prediction taps that areoutput from the blocking circuit 21 are supplied to the prediction tapsmemory 25. The prediction taps memory 25 stores, as learning data, theprediction taps that are supplied from the blocking circuit 21 at anaddress corresponding to the class that is supplied from the class codegeneration circuit 24.

On the other hand, the teacher data extraction circuit 27 extracts an HDpixel as the subject pixel from an HD picture supplied thereto, andsupplies it to the teacher data memory 26 as teacher data. The teacherdata memory 26 stores the teacher data that is supplied from the teacherdata extraction circuit 27 at an address corresponding to the class thatis supplied from the class code generation circuit 24.

Thereafter, similar processes are executed until all HD pixelsconstituting the HD picture that is prepared for the learning in advanceare employed as the subject pixel.

As a result, SD pixels and HD pixels that are in the positionalrelationships described above in connection with FIGS. 3 and 4 with thesubject pixel (i.e., SD pixels and HD pixels constituting predictiontaps) when the HD pixel that is stored at an address the teacher datamemory 26 is employed as the subject pixel (i.e., SD pixels and HDpixels constituting prediction taps) are stored, as learning data x, atthe same address of the prediction taps memory 25 as the address of theteacher data memory 26.

Then, an operation circuit 29 reads out, from the prediction taps memory25 and the teacher data memory 26, SD pixels and HD pixels constitutingprediction taps as learning data and HD pixels as teacher data that arestored at the same addresses, respectively. The operation circuit 29calculates prediction coefficients that minimize errors betweenprediction values and the teacher data by a least squares method byusing those pixels. That is, the operation circuit 29 establishes normalequations of Formula (18) for each class and determines predictioncoefficients for each class by solving the normal equations.

The prediction coefficients for the respective classes that have beendetermined by the operation circuit 29 in the above manner are stored inthe prediction coefficients memory 14 (see FIG. 5) together with theclasses.

In the above learning process, there may occur a class with which normalequations are not obtained in a number necessary for determiningprediction coefficients. For such a class, for example, predictioncoefficients that are obtained by establishing normal equations bydisregarding classes and solving those may be employed as what is calleddefault prediction coefficients.

While in the case of FIG. 5 HD pixels constituting an HD picture forlearning are used not only as HD pixels to be used as teacher data butalso as HD pixels to be used as learning data, HD pixels to used aslearning data may be generated from SD pixels for learning.

FIG. 7 shows an example configuration of a second embodiment of alearning apparatus. In FIG. 7, components having the correspondingcomponents in the learning apparatus of FIG. 5 are given the samereference numerals as the latter.

An HD picture for learning are supplied to a frame memory 31 capable ofstoring an HD picture of one frame or more and stored there. HD picturewrite and read operations on the frame memory 31 are controlled by acontrol section 37. The frame memory 31 stores an HD picture forlearning and reads out a stored HD picture for learning under thecontrol of the control section 37.

An HD picture that has been read out from the frame memory 31 issupplied to the vertical decimation filter 11 and a learning section 36.The vertical decimation filter 11 and the horizontal decimation filter12 that is provided downstream of it generate an SD picture for learningfrom the HD picture for learning in the same manner as described abovein connection with FIG. 5, and supply those to a switch 32 and thelearning section 36.

The switch 32 selects one of terminals 32 a and 32 b under the controlof the control section 37. Specifically, the switch 32 selects theterminal 32 a when the HD picture that is stored in the frame memory 31is read out first. Therefore, in this case, an SD picture for learningis supplied to a linear interpolation filter 34 via the switch 32 andthe terminal 32 a. The linear interpolation filter 34 generates apicture having the same number of pixels as the HD picture for learningfrom the SD picture for learning by, for instance, executing a linearinterpolation process on the SD picture for learning. Unlike theadaptive process, the linear interpolation process that is executed inthe linear interpolation filter 34 cannot reproduce high-frequencycomponents that are included in the original picture (in this case, theHD picture for learning). The picture having the same number of pixelsas the HD picture for learning, that is output from the linearinterpolation filter 34 will be hereinafter referred to as aninterpolation HD picture, where appropriate.

The interpolation HD picture that is output from the linearinterpolation filter 34 is supplied to a terminal 33 a. The controlsection 37 controls the switches 32 and 33 so that the switch 33 selectsthe terminal 33 a when the switch 32 selects the terminal 32 a and thatthe switch 33 selects the terminal 33 b when the switch 32 selects theterminal 32 b. Therefore, in the case being considered, since the switch33 selects the terminal 33 a, the interpolation HD picture is suppliedto the learning section 36.

The learning section 36 determines, for each class, predictioncoefficients that decrease errors of a picture obtained by a linearcombination of the prediction coefficients with respect to the HDpicture for learning that is supplied from the frame memory 31 byestablishing and solving normal equations of the above Formula (18)using the SD picture for learning that is supplied from the horizontaldecimation filter 12 and the interpolation HD picture that is suppliedfrom the linear interpolation filter 34. The prediction coefficients forthe respective classes are supplied to the prediction coefficientsmemory 14 together with the classes and stored there.

When the prediction coefficients for the respective classes have beenstored in the prediction coefficients memory 14, the control section 37reads out those prediction coefficients and supplies those to a pictureconversion apparatus 35. Configured in the same manner as the pictureconversion apparatus of FIG. 1 or 9, the picture conversion apparatus 35stores the prediction coefficients for the respective classes that aresupplied from the control section 37 in the prediction coefficientsmemory 5. When the HD picture is read out first from the frame memory31, no prediction coefficients are stored in the picture conversionapparatus 35 and hence an adaptive process cannot be executed.Therefore, when the HD picture is read out first from the frame memory31, an interpolation HD picture is generated by the linear interpolationfilter 34 and the learning section 36 determines prediction coefficientsusing the interpolation HD picture.

Thereafter, the control section 37 controls the frame memory 31 so as tostart a second read operation for reading out the HD picture forlearning stored therein. In the second and following read operations forreading out the HD picture stored therein, the control section 37controls the switches 32 and 33 so that they select the terminals 32 band 33 b, respectively.

As a result, the HD picture that has been read out from the frame memory31 is not only supplied to the learning section 36 but also supplied toit, after being converted into an SD picture for learning, via thevertical decimation filter 11 and the horizontal decimation filter 12.

In this case, furthermore the SD picture for learning that is output viathe vertical decimation filter 11 and horizontal decimation filter 12 isnot only supplied to the learning section 36 but also supplied to thepicture conversion apparatus 35 via the switch 32 that is selecting theterminal 32 b. The picture conversion circuit 35 executes an adaptiveprocess on the SD picture for learning by using that predictioncoefficients for the respective classes that have been set in theprediction coefficients memory 5 by the control section 37, and therebygenerates a picture having the same number of pixels as the HD picturefor learning.

The picture generated by the picture conversion apparatus 35 is suppliedto the terminal 33 b and hence supplied to the learning section 36 viathe switch 33 that is selecting the terminal 33 b.

The learning section 36 determines, for each class, predictioncoefficients that decrease errors of a picture obtained by a linearcombination of the prediction coefficients with respect to the HDpicture for learning that is supplied from the frame memory 31 byestablishing and solving normal equations of the above Formula (18)using the SD picture for learning that is supplied from the horizontaldecimation filter 12 and the picture that is supplied from the pictureconversion apparatus 35.

Therefore, in the second and following read operations for reading theHD picture from the frame memory 31, prediction coefficients for therespective classes are learned by using, as learning data, instead of alinear HD picture, a picture (hereinafter referred to as an adaptiveprocess picture where appropriate) that has the same number of pixels asthe HD picture for learning and is obtained by an adaptive processexecuted on an SD picture for learning in the picture conversionapparatus 35.

The prediction coefficients for the respective classes that have beenobtained by the learning section 36 are supplied to the predictioncoefficients memory 14 together with the classes and stored there (theyoverwrite the previously stored prediction coefficients for therespective classes).

When prediction coefficients for the respective classes have been storedin the prediction coefficients memory 14, as described above the controlsection 37 reads out those prediction coefficients for the respectiveclasses, and supplies and stores those to and in the picture conversionapparatus 35 that is configured in the same manner as the pictureconversion apparatus of FIG. 1 or 9. Further, the control section 37controls the frame memory 31 so as to start reading out the HD picturefor learning stored therein. Similar processes are repeated thereafter.

When the number of repetition times of operations for reading the HDpicture stored in the frame memory 31 has reached a predetermined numberor the error of an adaptive process picture that is output from thepicture conversion apparatus 35 with respect to the HD picture stored inthe frame memory 31 has become smaller than or equal to a predeterminedvalue, the control section 37 finishes the processes of the respectivecomponents constituting the apparatus with a judgment that predictioncoefficients suitable for converting an SD picture to an HD picture havebeen stored. For example, if it is found that a picture similar to theHD picture for learning has been obtained, by an operator who visuallychecks adaptive process pictures obtained by the picture conversionapparatus 35, the processes may be finished manually.

FIG. 8 shows an example configuration of the learning section 36 shownin FIG. 7. In FIG. 8, components having the corresponding components inthe learning section 13 shown in FIG. 5 are given the same referencenumerals as the latter. That is, the learning section 36 is configuredin the same manner as the learning section 13 of FIG. 6 except that thedelay line 28 is supplied with an interpolation HD picture that isoutput from the linear interpolation filter 34 or an adaptive processpicture that is output from the picture conversion apparatus 35 ratherthan an HD picture for learning.

Therefore, in the learning section 36, when an HD picture is first readout from the frame memory 31 (see FIG. 7), the blocking circuits 21 and22 form prediction taps and class taps, respectively, from an SD picturefor learning and an interpolation HD picture. When the HD picture isread out from the frame memory 31 at a second or later time, theblocking circuits 21 and 22 form prediction taps and class taps,respectively, from an SD picture for learning and an adaptive processpicture. The learning section 36 determines prediction coefficients forthe respective classes in the same manner as the learning section 13 ofFIG. 6 except for the above operations.

In a simulation that was conducted by the present inventor, predictioncoefficients capable of producing an HD picture of better picturequality were obtained by the learning apparatus of FIG. 5 than by thelearning apparatus of FIG. 7.

The picture conversion apparatus and the learning apparatuses to whichthe present invention is applied have been described above. This type ofpicture conversion apparatus can be applied, in addition to the case ofconverting an SD picture into an HD picture, to any cases of convertinga picture having a small number of pixels to a picture having a largenumber of pixels such as a case of an interlace-scanned picture intowhat is called a progressive picture, and a case of enlarging a picture,and a case of a NTSC format to PAL format. Also, this type of pictureconversion apparatus can be applied to any format conversion regardlessof converting a picture having a small number of pixels to a picturehaving a large number of pixels.

Although in the above embodiments the processes are executed on aframe-by-frame basis, they may be executed in different manners, forinstance, on a field-by-field basis.

The invention can be applied to both of a moving picture and a stillpicture.

Although in the above embodiments an HD picture of a frame preceding thesubject frame by one frame is used to form prediction taps and classtaps, an HD picture of some other frame may be used such as the subjectframe or a frame succeeding the subject frame by one frame. However, inthe latter cases, for example, it becomes necessary to provide twosystems of picture conversion apparatuses of FIG. 1 or providing a delayline for time adjustment upstream of or downstream (at the front stageor the back stage) of the picture conversion apparatus.

Similarly, an SD picture of a frame preceding or succeeding the subjectframe by one frame can be used in addition to the subject frame to formprediction taps and class taps.

The relationships between SD pixels and HD pixels are not limited to theones shown in FIG. 3. The present invention may be represented as aprogram that can be executed on a general computer.

In the picture conversion apparatus and the picture conversion method towhich the present invention is applied, prediction taps are formed froma first picture and a picture obtained by an adaptive process, and theadaptive process is executed by using the prediction taps and predictioncoefficients that are adapted to the prediction taps. Therefore, apicture having better picture quality can be obtained.

In the learning apparatus and the learning method to which the presentinvention is applied, a first picture for learning is generated bydecreasing the number of pixels of a second picture for learning. Andthen prediction coefficients that decrease an error, with respect to thesecond picture for learning, of a picture obtained by a linearcombination of the first and second pictures for learning and theprediction coefficients, are determined. Therefore, predictioncoefficients for obtaining a picture having better picture quality canbe obtained.

In the learning apparatus and the learning method to which the presentinvention is applied, a first picture for learning is generated bydecreasing the number of pixels of a second picture for learning, and anadaptive process picture is output by executing an adaptive process onthe first picture for learning. Then, prediction coefficients thatdecrease an error, with respect to the second picture for learning, of apicture obtained by a linear combination of the first picture forlearning and the adaptive process picture and the predictioncoefficients, are determined. Further, the adaptive process fordetermining an adaptive process picture is again executed by using thethus-determined prediction coefficients. Therefore, predictioncoefficients for determining an adaptive process picture having betterpicture quality can be obtained.

Having now fully described the invention, it will be apparent to one ofordinary skill in the art that many changes and modifications can bemade thereto without departing from the spirit and scope of theinvention as set forth herein.

What is claimed is:
 1. An apparatus for converting a first picturecomprising pixels into a second picture comprising pixels, the secondpicture being converted by executing on the first picture an adaptiveprocess that determines prediction values of the second picture by usinga number of pixel values of the first picture as prediction taps and anumber of prediction coefficients that are adapted to the first picture,said apparatus comprising: a prediction taps forming circuit for forminga number of prediction taps from the first picture and a pictureobtained by the adaptive process; and an executing circuit for executingthe adaptive process by using the formed number of prediction taps and anumber of prediction coefficients that are adapted to the predictiontaps.
 2. An apparatus according to claim 1, further comprising: a classtaps forming circuit for forming a number of class taps from the firstpicture and the picture obtained by the adaptive process; and aclassifying circuit for classifying the number of class taps todetermine a class; wherein the executing circuit executes the adaptiveprocess by using the formed number of prediction taps and the number ofprediction coefficients corresponding to the class.
 3. An apparatusaccording to claim 2, wherein the executing circuit reads out the numberof prediction coefficients from a memory in response to the class andcalculating the prediction value of the second picture by using theformed number of prediction taps and the read number of predictioncoefficients, the memory storing a number of prediction coefficients forrespective classes.
 4. An apparatus according to claim 3, wherein thenumber of prediction coefficients for respective classes are generatedin advance by using a second learning picture having a qualitycorresponding to a quality of the second picture.
 5. An apparatusaccording to claim 4, wherein the number of prediction coefficients forrespective classes are chosen so as to minimize an error between thesecond learning picture and a picture predicted from a first learningpicture and the second learning picture.
 6. An apparatus according toclaim 4, wherein the number of prediction coefficients for respectiveclasses are selected so as to minimize an error between the secondlearning picture and a picture predicted from a first learning pictureand an adaptive processed second picture, the adaptive processed secondpicture being obtained by executing the adaptive process on the firstlearning picture.
 7. An apparatus for converting a first picturecomprising pixels into a second picture comprising pixels, the secondpicture being converted by executing on the first picture an adaptiveprocess that determines prediction values of the second picture by usinga number of pixel values of the first picture as prediction taps and anumber of prediction coefficients that are adapted to the first picture,said apparatus comprising: a class taps forming circuit for forming anumber of class taps from the first picture and a picture obtained bythe adaptive process; a classifying circuit for classifying the numberof class taps to determine a class; a prediction taps forming circuitfor forming a number of prediction taps from the first picture; and anexecution circuit for executing the adaptive process by using the formednumber of prediction taps and a number of prediction coefficientscorresponding to the class.
 8. An apparatus according to claim 7,wherein the execution circuit reads out the number of predictioncoefficients from a memory in response to the class and calculating theprediction value of the second picture by using the formed number ofprediction taps and the read number of prediction coefficients, thememory storing a number of prediction coefficients for respectiveclasses.
 9. A method for converting a first picture comprising pixelsinto a second picture comprising pixels, the second picture beingconverted by executing on the first picture an adaptive process thatdetermines prediction values of the second picture by using a number ofpixel values of the first picture as prediction taps and a number ofprediction coefficients that are adapted to the first picture, saidmethod comprising the steps of: forming a number of prediction taps fromthe first picture and a picture obtained by the adaptive process; andexecuting the adaptive process by using the formed number of predictiontaps and a number of prediction coefficients that are adapted to theprediction taps.
 10. A method according to claim 9, further comprisingthe steps of: forming a number of class taps from the first picture andthe picture obtained by the adaptive process; and classifying the numberof class taps to determine a class; wherein the step of executing theadaptive process executes the adaptive process by using the formednumber of prediction taps and the number of prediction coefficientscorresponding to the class.
 11. A method according to claim 10, whereinthe step of executing the adaptive process further comprises the stepsof reading out the number of prediction coefficients from a memory inresponse to the class and calculating the prediction value of the secondpicture by using the formed number of prediction taps and the readnumber of prediction coefficients, the memory storing a number ofprediction coefficients for respective classes.
 12. A method accordingto claim 11, wherein the number of prediction coefficients forrespective classes are generated by using a second learning picturehaving a quality corresponding to a quality of the second picture.
 13. Amethod according to claim 12, wherein the number of predictioncoefficients for respective classes are those so as to minimize an errorbetween the second learning picture and a picture predicted from a firstlearning picture and the second learning picture.
 14. A method accordingto claim 12, wherein the number of prediction coefficients forrespective classes are selected so as to minimize an error between thesecond learning picture and a picture predicted from a first learningpicture and an adaptive processed second picture, the adaptive processedsecond picture being obtained by executing the adaptive process on thefirst learning picture.
 15. A method for converting a first picturecomprising pixels into a second picture comprising pixels, the secondpicture being converted by executing on the first picture an adaptiveprocess that determines prediction values of the second picture by usinga number of pixel values of the first picture as prediction taps and anumber of prediction coefficients that are adapted to the first picture,said method comprising the steps of: forming a number of class taps fromthe first picture and a picture obtained by the adaptive process;classifying the number of class taps to determine a class; forming anumber of prediction taps from the first picture; and executing theadaptive process by using the formed number of prediction taps and anumber of prediction coefficients corresponding to the class.
 16. Amethod according to claim 15, wherein the step of executing the adaptiveprocess reads out the number of prediction coefficients from a memory inresponse to the class and further comprising the step of calculating theprediction value of the second picture by using the formed number ofprediction taps and the read number of prediction coefficients, thememory storing a number of prediction coefficients for respectiveclasses.
 17. An apparatus for converting a first picture comprisingpixels into a second picture comprising pixels, the second picture beingconverted by executing on the first picture an adaptive process thatdetermines prediction values of the second picture by using a number ofpixel values of the first picture as prediction taps and a number ofprediction coefficients that are adapted to the first picture, saidapparatus comprising: means for forming a number of prediction taps fromthe first picture and a picture obtained by the adaptive process; andmeans for executing the adaptive process by using the formed number ofprediction taps and a number of prediction coefficients that are adaptedto the prediction taps.
 18. An apparatus according to claim 17, furthercomprising: means for forming a number of class taps from the firstpicture and a picture obtained by the adaptive process; and means forclassifying the number of class taps to determine a class; wherein theexecuting means executes the adaptive process by using the formed numberof prediction taps and the number of prediction coefficientscorresponding to the class.
 19. An apparatus according to claim 18,wherein the executing means reads out the number of predictioncoefficients from a memory in response to the class and calculating theprediction value of the second picture by using the formed number ofprediction taps and the read number of prediction coefficients, thememory storing a number of prediction coefficients for respectiveclasses.
 20. An apparatus according to claim 19, wherein the number ofprediction coefficients for respective classes are generated in advanceby using a second learning picture having a quality corresponding to aquality of the second picture.
 21. An apparatus according to claim 20,wherein the number of prediction coefficients for respective classes arethose so as to minimize an error between the second learning picture anda picture predicted from a first learning picture and the secondlearning picture.
 22. An apparatus according to claim 20, wherein thenumber of prediction coefficients for respective classes are selected soas to minimize an error between the second learning picture and apicture predicted from a first learning picture and a adaptive processedsecond picture, the adaptive processed second picture being obtained byexecuting the adaptive process on the first learning picture.
 23. Anapparatus for converting a first picture comprising pixels into a secondpicture comprising pixels, the second picture being converted byexecuting on the first picture an adaptive process that determinesprediction values of the second picture by using a number of pixelvalues of the first picture as prediction taps and a number ofprediction coefficients that are adapted to the first picture, saidmethod comprising the steps of: means for forming a number of class tapsfrom the first picture and a picture obtained by the adaptive process;means for classifying the number of class taps to determine a class;means for forming a number of prediction taps from the first picture;and means for executing the adaptive process by using the formed numberof prediction taps and a number of prediction coefficients correspondingto the class.
 24. An apparatus according to claim 23, wherein theexecuting means reads out the number of prediction coefficients from amemory in response to the class and calculating the prediction value ofthe second picture by using the formed number of prediction taps and theread number of prediction coefficients, the memory storing a number ofprediction coefficients for respective classes.